System and method for assisting a user in locating physical objects while the user is in a virtual reality environment

ABSTRACT

A method and apparatus is disclosed for assisting a user, wearing a head mounted display (HMD) that covers a user&#39;s field of vision and has a tracker providing information regarding the position and orientation of the HMD, in locating a physical controller located on a physical base station. A processor causes the HMD to display a virtual world, including a virtual representation of the physical base station and physical controller along with a virtual hand that helps guide the user to the physical base station to allow the user to pick up the physical controller. Another embodiment allows an area in the physical world to be defined within which the user should remain, for example to avoid physical obstacles. The processor causes the HMD to display a warning, such as a virtual fence, to alert the user if the user approaches to within a preselected distance of the boundary.

FIELD OF THE INVENTION

The present invention relates generally to virtual reality devices, andmore particularly to controllers for use in a virtual reality context.

BACKGROUND OF THE INVENTION

Virtual reality is often used to describe a wide variety of applicationscommonly associated with immersive, highly visual, computer-simulatedenvironments that can simulate a user's physical presence in places inthe real world or imagined worlds. While virtual reality can recreate anumber of sensory experiences, the senses most commonly used today tocreate virtual reality appear to be sight and sound.

One method of presenting a virtual world to a user that is commonlyidentified with virtual reality and presently in use is through the useof a visor or helmet containing a video display which encompasses partor all of a user's field of view and presents computer generated imagesrepresenting the virtual reality environment, or “virtual world,” to theuser. Such a device is often referred to as a head-mounted display, orHMD. One type of HMD presently available is the Oculus Rift from OculusVR, now owned by Facebook.

Typically a HMD covers the user's eyes, so that the user sees only thevirtual world while wearing the HMD and is thus unable to see the actualphysical world around the user while in the virtual world. For thisreason, the use of a HMD can create some issues when it is necessary ordesirable that the user interact with, or sometimes not interact with,the real world. Since the user cannot see the real environment, the usermay not know what actions by the user should occur in the real world,and what actions may have detrimental consequences in the real world.

One instance of such an issue may arise when the user wishes to be ableto use some type of input device while in the virtual world. Forexample, a user may wish to use a handheld controller, such as might beused to play a video game, while in a virtual world, and perhaps eventwo controllers, one for each hand. However, since the user cannot seethe physical world once he or she is wearing the HMD, it can bedifficult for the user to locate and pick up the controller afterputting on the HMD, as the user must blindly fumble around the physicalenvironment to locate the controller(s).

Alternatively, it can be difficult to pick up and put on a HMD if theuser has already picked up and is holding a controller in his or herhand, and even more so if there are two controllers. In this case, theuser may drop one or more of the controller(s) or HMD, resulting inpossible damage to the devices.

Another issue with respect to the user's interactions with the physicalworld is that it is difficult for the user to maintain awareness of hisor her location in the physical world while the user sees only thevirtual world. It may be desirable that the user remain within aselected area, for example, if there are physical objects in thevicinity of the user. The user may wish to be able to move within thevirtual world by, for example, taking a step or moving an arm; ofcourse, any such movement also results in the user moving in thephysical world. When playing a conventional video game on a television,for example, even when using a controller which responds to the user'smotion the user is able to see nearby objects, and avoid moving thecontroller in a way which causes the user to collide with such objects.However, when using a HMD the user is unable to see those nearby objectswhile in the virtual world, and may thus not be aware of impendingcollisions.

It is thus desirable to find a solution that will allow a user in avirtual world presented by a HMD to be aware of the user's physicalsurroundings to a sufficient degree to allow the user to pick uphandheld controllers, and to remain in a selected area, for example toavoid collisions with surrounding physical objects.

SUMMARY OF THE INVENTION

A system and method is disclosed which provides visual assistance to auser in a virtual reality environment in locating physical objects whichthe user is unable to see, specifically a physical controller that auser may use to control certain actions or events in the virtual worldand which is located on a physical base station. Another system andmethod is disclosed which provides visual assistance to a user in avirtual reality environment in remaining within a selected area, forexample to avoid physical objects in the user's vicinity which the useris unable to see while in the virtual world.

One embodiment discloses a method of assisting a user wearing ahead-mounted display (HMD) in locating a physical handheld controllerlocated on a physical base station, the HMD covering the user's field ofvision and having a HMD tracker which provides information regarding theposition and orientation of the HMD, the method comprising: generatingand outputting, by a processor, instructions to the HMD to display onthe HMD a virtual world; determining, by the processor, the position andorientation of the HMD in the physical world using information from theHMD tracker; generating and outputting, by the processor, instructionsto the HMD for display on the HMD a virtual base station, a virtualcontroller and a virtual hand in the virtual world, the virtual basestation and virtual controller in apparent locations corresponding toactual physical locations of the physical base station and physicalcontroller and the virtual hand in an assumed apparent location;determining, by the processor, based upon the position and orientationof the HMD, that the user has moved toward the physical base station inthe physical world; generating and outputting, by the processor,instructions to the HMD for display on the HMD the virtual base station,virtual controller, and virtual hand, with the virtual hand closer tothe virtual base station than the assumed apparent location;determining, by the processor, that the physical controller has beenremoved from the physical base station; and generating and outputting,by the processor, based upon the determination that the physicalcontroller has been removed from the physical base station, instructionsto the HMD to display on the HMD a virtual hand in the virtual world,without displaying the virtual base station or the virtual controller.

Another embodiment discloses system for use with a head-mounted display(HMD), the HMD covering a user's field of vision and having a HMDtracker which provides information regarding the position andorientation of the HMD, comprising: a physical base station; a physicalhandheld controller located on the physical base station; and aprocessor configured to: generate and output instructions to the HMD todisplay on the HMD a virtual world; determine the position andorientation of the HMD in the physical world using information from theHMD tracker; generate and output instructions to the HMD for display onthe HMD a virtual base station, a virtual controller and a virtual handin the virtual world, the virtual base station and virtual controller inapparent locations corresponding to actual physical locations of thephysical base station and physical controller and the virtual hand in anassumed apparent location; determine, based upon the position andorientation of the HMD, that the user has moved toward the physical basestation in the physical world; generate and output instructions to theHMD for display on the HMD the virtual base station, virtual controller,and virtual hand, with the virtual hand closer to the virtual basestation than the assumed apparent location; determine that the physicalcontroller has been removed from the physical base station; and generateand output instructions to the HMD, based upon the determination thatthe physical controller has been removed from the physical base station,to display on the HMD the virtual hand in the virtual world, withoutdisplaying the virtual base station or the virtual controller.

Still another embodiment discloses a non-transitory computer-readablemedium having embodied thereon a program, the program being executableby a processor to perform a method of assisting a user wearing ahead-mounted display (HMD), the HMD covering the user's field of visionand having a tracker which tracks the position and orientation of theHMD, in locating a physical handheld controller that is initially placedon a physical base station, the method comprising the steps of:generating and outputting, by a processor, instructions to the HMD todisplay on the HMD a virtual world; determining, by the processor, theposition and orientation of the HMD in the physical world usinginformation from the HMD tracker; generating and outputting, by theprocessor, instructions to the HMD for display on the HMD a virtual basestation, a virtual controller and a virtual hand in the virtual world,the virtual base station and virtual controller in apparent locationscorresponding to actual physical locations of the physical base stationand physical controller and the virtual hand in an assumed apparentlocation; determining, by the processor, based upon the position andorientation of the HMD, that the user has moved toward the physical basestation in the physical world; generating and outputting, by theprocessor, instructions to the HMD for display on the HMD the virtualbase station, virtual controller, and virtual hand, with the virtualhand closer to the virtual base station than the assumed apparentlocation; determining, by the processor, that the physical controllerhas been removed from the physical base station; and generating andoutputting, by the processor, based upon the determination that thephysical controller has been removed from the physical base station,instructions to the HMD to display on the HMD a virtual hand in thevirtual world, without displaying the virtual base station or thevirtual controller.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B are illustrations of a user wearing a head mounteddisplay (HMD).

FIG. 2 is an illustration of a generated virtual world including avirtual base station and virtual controllers as it might be seen by auser wearing a HMD.

FIG. 3 is an illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD.

FIG. 4 is another illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD as the user approaches a physical basestation.

FIG. 5 is another illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD after the user has removed a physicalcontroller from the physical base station.

FIG. 6 is another illustration of a generated virtual world includingvirtual hands as it might be seen by a user wearing a HMD after the userhas removed both physical controllers from the physical base station.

FIG. 7 is another illustration of a generated virtual world including avirtual base station, virtual controllers and virtual hands as it mightbe seen by a user wearing a HMD after instructing the processor todisplay the virtual base station and virtual controllers.

FIG. 8 is an illustration of a generated virtual world including avirtual hand holding a virtual sword as it might be seen by a userwearing a HMD after the user has activated a handheld controller in aparticular situation.

FIG. 9 is a flowchart of a method of assisting a user wearing a HMD inlocating a physical handheld controller that is initially placed on aphysical base station according to one embodiment.

FIG. 10 is an illustration of a user and a virtual hand in a virtualworld as the virtual hand might be seen by a user wearing a HMD.

FIG. 11 another illustration of a user, a virtual hand and a visualwarning in a virtual world as virtual hand and visual warning might beseen by a user wearing a HMD the user approaches a boundary defining aselected area in the physical world.

FIG. 12 is an illustration of a selected area in one embodiment.

FIG. 13 is an illustration of a selected area in another embodiment.

FIG. 14 is an illustration of a selected area in still anotherembodiment.

FIG. 15 is a flowchart of a method of assisting a user wearing a HMD inremaining in a selected area according to one embodiment.

DETAILED DESCRIPTION OF THE INVENTION

A method and apparatus is disclosed for assisting a user in locatingphysical objects while the user is wearing a head mounted display (HMD)that covers a user's field of vision and has a tracker allows aprocessor to track the position and orientation of the HMD, the objectsbeing specifically one or more physical controllers that a user may useto control certain actions or events in the virtual world and which arelocated on a physical base station. Another described method andapparatus assists such a user in avoiding physical obstacles in theuser's vicinity. Since the user wearing the HMD sees only a virtualworld, the user is unable to see such physical objects or obstacles.

In one embodiment, when a user activates a display of a virtual worldinto which the user wishes to enter, for example, through the display ofa HMD, a processor tracks the position and orientation of a tracker onthe HMD. The processor also instructs that display to include in thedisplay of the virtual world a virtual representation of the physicalbase station and the physical controller, along with a virtual hand thathelps guide the user to the physical base station so as to allow theuser to pick up the physical controller. Once the physical controllerhas been picked up and the user has moved a certain distance from thephysical base station, a virtual hand may be shown without the virtualcontroller to heighten the sense of realism in the virtual world. Thisallows the user to locate and pick up the physical controller(s) by thesight provided in the virtual world, rather than having to feel aroundto locate the controller(s) by touch.

In another embodiment, a boundary of a selected area is determined, forexample, a “safe area” which the user or another person has determinedis clear of physical obstacles in the physical world so that the usermay move freely without coming into contact with such physicalobstacles. The processor similarly tracks the position and orientationof the HMD tracker, and possibly the position and orientation of one ormore trackers in the physical controllers, and generates instructions tothe HMD to include in a display of a virtual world a visual warning suchas a grid or virtual fence when the user comes within a preselecteddistance of the boundary, thus providing a warning that the user isabout to move out of the selected area.

FIG. 1A shows a user 102 in the real world wearing a HMD 106; HMD 106covers the user's eyes and presents the user with a representation of avirtual world, while preventing the user from seeing the real, orphysical, world.

Attached to or incorporated in the HMD 106 is a tracker 108, which canbe used to determine the position and orientation of the headset 104,and thus of the head of user 102, by, for example, sensing anelectromagnetic field generated from a local base station, video gameconsole or other apparatus (not shown). The tracker 108 providesinformation about the sensed magnetic field to a processor (also notshown), which derives the position and orientation of the tracker 108,and thus HMD 106, relative to a source of the electromagnetic field fromsuch information.

Such trackers, and the processors which determine the position andorientation of such trackers, are well known to those of skill in theart. One such example, U.S. Pat. No. 4,737,794, teaches a “method andapparatus for determining remote object orientation and position with anelectromagnetic coupling.” It shows a plurality of radiating antennaslocated at a source to provide a plurality of electromagnetic fieldsspanning three dimensional space and defining a source referencecoordinate frame, and a plurality of receiving antennas located on anobject to be tracked for receiving that field. A processor receives theoutputs from the receiving antennas and converts the received componentsof the transmitted electromagnetic fields into remote object positionand orientation relative to the source reference coordinate frame. Theantennas may be, for example, dipole antennas, loop antennas, or coilantennas.

Another such example are light-emitting trackers which can be seen by,and therefore provide position and orientation information to, a cameramounted on the base station or elsewhere. Other forms of trackers,including those incorporating inertial sensors (e.g., accelerometersand/or gyroscopes), can likewise be used in the present approach. Aswould be understood by one of ordinary skill in the art in light of theteachings herein, essentially any type of tracker (e.g., magnetic,optical, electrical, sonar, radar, ultrasonic, or any combinationthereof) which can provide information regarding its position (and, insome cases, orientation, if needed or desired) can be used in thepresent approach. Such techniques are all contemplated as being withinthe described system and method, including techniques which can onlyprovide position information, and not orientation, even though the belowdiscussion generally refers to electromagnetic tracking.

FIG. 1A also shows the user 102 holding a controller 110 which the usermay pick up with his or her hand(s), and which allows the user toperform various functions in the virtual world. Various devices suitablefor use as controller 110 will also be known to those of skill in theart. In addition to tracker 108 in HMD 106, handheld controller 110 mayalso have a tracker, from which a processor can derive the position andorientation of handheld controller 110 relative to the base station, byany of the tracking techniques described above. Techniques foridentifying multiple trackers and distinguishing the position andorientation of each are well known to those of skill in the art.

FIG. 1B is a representation of the user 102 of FIG. 1A as the user 102sees the virtual world, showing the field of view 112 that may bepresented to the user by the HMD 106. As will be known to those of skillin the art, a processor (not shown) generates instructions that causeHMD 106 to present a view to user 102 that is appropriate given theposition and orientation of user 102's head and any actions taken withcontroller 110, so that what is within field of view 112 will change asuser 102 moves his or her head and uses controller 110.

In the illustration of FIG. 1B, the user is seeing the virtual world aspresented on HMD 106. In FIG. 1B the user also sees a virtual object 114in his or her hand, here a virtual gun, rather than the controller 110that the user is actually holding as in FIG. 1A. As will be explainedbelow, the user has presumably used the controller to interact with thevirtual world to “pick up” the virtual gun 114.

In some embodiments, the controller 110 may be placed or “docked” in abase station or video game console (not shown). In some embodiments, thebase station or console may contain the processor as well as the othercomponents needed to determine the position and orientation of thetrackers in the HMD and controllers. For example, the base station maycontain the electromagnetic field source for electromagnetic trackers, acamera for optical detection of certain types of trackers, or otherelements needed to determine the position and orientation of other typesof trackers.

In other embodiments, the base station may be only a cradle or dock uponwhich the controller may rest, perhaps with a groove or contour thatconforms to the shape of the controller, with the processor and/or othercomponents located elsewhere. Alternatively, some components may be inthe base station and some components located elsewhere. If componentssuch as the electromagnetic field source or camera used to determine theposition of the HMD and controller trackers relative to the base stationare not located in the base station, it will be desirable for the basestation to also contain a tracker similar to those used in the HMD andcontrollers, so that the position of the base station is known and thusthe positions of the HMD and controllers relative to the base stationcan be determined.

In the description herein, it is assumed that the position of the HMDand controller trackers relative to the base station may be determinedby any of the tracking means described above, regardless of whether theprocessor and other components needed for determining those relativepositions are located in the base station or elsewhere.

In one embodiment, controller 110 contains a rechargeable battery, andcircuitry in the base station may charge the battery when the controlleris located in the base station. In other embodiments, there may be twocontrollers 110, where it is intended that there is one controller 110for each hand of user 102.

As above, with a HMD 106 which covers the eyes of user 102, once user102 has donned the HMD 106, user 102 can no longer see the surroundingphysical world, including the controller 110 or a physical base stationin which the controller 110 may be placed. User 102 is thus unable todetermine where the controllers are in the physical world in order topick them up other than perhaps by blindly feeling around the area inwhich the user believes the base station might be located.

For this reason, rather than forcing the user to locate the controllerby touch, one embodiment provides a visual representation in the virtualworld of a virtual base station and virtual controller so that the usermay find and pick up the virtual controller by sight in the virtualworld, thus picking up the physical controller without being able to seethe actual physical controller due to the HMD 106 presenting the userwith only a representation of the virtual world.

FIG. 2 shows what a user such as user 102 might see in the virtual worlddisplayed in the HMD 106 in one embodiment. In the illustratedembodiment, the processor has generated instructions to the display ofHMD 106 to create a picture of a virtual world as if the user is lookingtoward a back wall 202, and also displays side walls 204, floor 206 andceiling 208. Also displayed is a virtual base station 210, in which aredocked two virtual controllers 212.

In one embodiment, the processor causes the HMD 106 to display virtualbase station 210 and virtual controllers 212 as having the same generalappearance as the actual physical base station and the physicalcontrollers that the user will pick up and use to control interactionswith the virtual world, and to appear to the user in an apparentlocation in the virtual world which corresponds to the actual locationof the physical base station and controllers in the real world. Thus,although the user is only seeing the virtual world, if the userapproaches the virtual base station 210, he or she will also beapproaching the physical base station in the real world.

As above, in some embodiments a user such as user 102 wears a HMD 106which has attached to it a tracker 108 which can track the position andorientation of the headset 104, and thus the head of user 102 by any ofthe tracking techniques above, such as with respect to anelectromagnetic field generated by a source located in the physical basestation or elsewhere. (Again, the base station should also have atracker if such components as the electromagnetic source or camera usedto track the HMD are not located in the base station.) Thus, as the userapproaches the virtual base station 210 presented on the display, theprocessor receiving information from tracker 108 (and from the basestation tracker where appropriate) can determine that the user is alsoapproaching the physical base station.

FIG. 3 shows what a user might see in the virtual world in oneembodiment as the user has approached the virtual base station 210 inthe virtual world and is looking down towards it, and is alsoapproaching the physical base station in the real world. Now theprocessor has generated instructions to alter the display to the user sothat the perspective has changed, and it appears as if the user iscloser to the virtual base station 210 and looking down towards it. Theback wall 202, side walls 204 and floor 206 are still displayed.

In addition, the instructions from the processor now also cause thedisplay to generate two virtual hands 214 (with a portion of theforearms) reaching toward the virtual base station 210. At this point,since the processor has no information about where the user's hands arein the physical world, the location of these virtual hands 214 isassumed from the position and orientation of the user's head asdetermined from the tracker, and are intended to provide guidance andassistance to the user in locating the physical base station andphysical controllers in the physical world.

As the user gets still closer to the physical base station and physicalcontrollers in the physical world, again as determined from the positionof the tracker of the HMD on the user's head, in one embodiment theprocessor will generate instructions to alter the display as shown inFIG. 4. Now the user appears to be closer to the virtual base station210, and the virtual hands 214 appear to be closer to the virtual basestation 210, than in FIG. 3, although again at this point the locationof virtual hands 214 is assumed from information provided by the trackeron the HMD on the user's head. Again, the virtual base station 210 andvirtual hands 214 assist the user in locating the physical base stationand physical controllers.

FIG. 5 shows what the user might see in the displayed virtual world inone embodiment after grasping one of the physical controllers andremoving it from the physical base station. The processor now generatesinstructions to display the virtual world much as it appears in FIG. 4,with the back wall 202, side walls 204, floor 206, and virtual basestation 210 displayed. However, as will be explained below, now theprocessor is able to determine that the user has removed the rightphysical controller from the physical base station in the physicalworld, and now displays the virtual right hand 214 as holding the rightvirtual controller 212 instead of the right virtual controller 212 beingdocked in the virtual base station as in FIG. 4.

As above, the physical controllers may also have magnetic trackerslocated within them, similar to the tracker in the headset. If so, theprocessor is now able to determine from information received from thetracker in the right physical controller where the user's hand holdingthat controller is located. The processor may then display the virtualright hand 214 in a location within the virtual world that appears tothe user to be where the physical right hand is actually located, ratherthan providing only an estimated location as before.

In one embodiment, the processor is able to detect that the electricalconnection between the right physical controller and the physical basestation has been broken, and thus determine that the right physicalcontroller has been removed from the physical base station. In anotherembodiment, by using information from the tracker in the right physicalcontroller to determine the position of the right physical controllerrelative to the physical base station, the processor is able todetermine that the right physical controller is no longer in thelocation in which it is docked in the physical base station.

At the point in time shown in FIG. 5, since the left physical controllerhas not been removed from the physical base station, the instructionsfrom the processor cause the left virtual controller 212 to still bedisplayed as docked in the virtual base station 210. Similarly, virtualhand 214 is still shown in an estimated location, since the processorstill has no information about where the user's physical left hand isactually located.

Once both controllers have been picked up, to enhance the realism of thevirtual world, it is desirable that the user no longer see the virtualbase station 210 or the virtual controllers 212, i.e., that theinstructions from the processor no longer cause the HMD to show them. Asdiscussed below with respect to FIG. 8, this can, for example, allow theprocessor to cause the HMD to display the user's virtual hands doingsomething in the virtual world, such as grasping a virtual object,rather than holding the virtual controllers (even though the user willof course continue to feel the physical controllers in his or herhands). FIG. 6 shows one embodiment of what might be displayed to a userin a displayed virtual world once both physical controllers have beenpicked up by the user. Instructions from the processor again cause thedisplay of the virtual world to appear much the same as it appears inFIG. 4 or 5, with the back wall 202, side walls 204, and floor 206displayed. However, now the processor has determined that the user hasremoved both physical controllers from the physical base station, it nowdisplays both virtual hands 214 without either the virtual controllers212 or the virtual base station 210 being displayed.

Since the processor is now able to determine from information receivedfrom the controller trackers in the physical controllers where theuser's hands holding the controllers are located, the processor may thendisplay both virtual hands 214 in apparent locations within the virtualworld that correspond to where the user's physical hands are actuallylocated, rather than providing only an estimated location for either orboth virtual hands 214 as before.

At some point, the user may wish to return the physical controllers tothe physical base station. In one embodiment, the user may cause theprocessor to display both the virtual base station 210 and the virtualcontrollers 212, by, for example, entering an appropriate command on acontroller, so that the user may see their locations in the virtualworld. FIG. 7 shows what a user might see once such a point is reached.The instructions from the processor to the HMD again cause the virtualworld to appear much the same as it appears in the previous FIGS. 4 to6, with the back wall 202, side walls 204, and floor 206 displayed. Now,however, the virtual base station 210 and virtual controllers 212 areagain displayed, with the user's two virtual hands 214 holding thevirtual controllers 212, as a user might see his or her hands in thereal world.

As with the virtual hands 214 in FIG. 6, the virtual hands 214 of FIG. 7are shown in apparent locations in the virtual world that correspond tothe actual locations of the user's physical hands in the physical world.Since the processor still has information from the trackers in thephysical controllers being held in the user's hands, the actual locationof the user's hands can still be determined and therefore the virtualhands can be shown by the HMD in such corresponding apparent locations.Since the virtual base station is also in an apparent location in thevirtual world that corresponds to the location of the physical basestation in the physical world, by appearing to place the virtualcontrollers 212 on the virtual base station 210 the user is guided toplace the physical controllers on the physical base station.

In some embodiments, it may be desirable to have the virtual basestation 210 and virtual controllers 212 continue to be displayed untilthe user has moved the physical controllers beyond some shortpreselected distance, for example, a few inches, away from the physicalbase station, and then displayed again when the user has approached towithin the preselected distance. This allows for added realism in thevirtual world, since the user will no longer see the virtual controllersall the time, while maintaining the convenience of seeing the virtualcontrollers near the virtual base station when the physical controllersare close to the physical base station. This may be particularly helpfulin returning the physical controllers to the physical base station, asexplained above. The preselected distance will typically bepre-programmed into the processor as part of the program which generatesand outputs the instructions to the HMD to display the virtual basestation and virtual controllers. Alternatively, the preselected distancemay be selectable by the user in some embodiments.

The processor can continue to generate instructions to the display inthe HMD to present the user with changes in the virtual world based uponactions that the user has performed with one or both of the physicalcontrollers. In a conventional video game, for example, a user may use acontroller to have a character on a television screen “pick up” a weapondisplayed on the screen. Similarly, with the present approach theprocessor can cause a HMD to display a virtual weapon while the user isin a virtual world. Then, upon the user's extending his or her physicalhand(s) in such a way as to cause the corresponding virtual hand(s) to“reach for” the displayed virtual weapon and the user's activating aproper command (e.g., by pressing a button on the controller), theprocessor can cause the HMD to display the user's virtual hand(s) 214 as“picking up” and holding that virtual weapon.

FIG. 8 shows one example of what a user might see in a virtual worldafter the user has extended the user's physical right hand to cause thevirtual right hand to extend toward a virtual sword displayed in thevirtual world on the HMD, and then entered the appropriate command topick up the virtual sword 802. The user's virtual right hand 214 isholding the virtual sword 802, although of course there is no physicalsword and the user's physical right hand is instead still holding thephysical controller. The processor can then display movements of thevirtual sword 802 based upon movements of the user's physical right handas determined from motion of the tracker in the physical controller.

In some embodiments, the processor can provide instructions to the HMDto make the virtual base station 210 always be displayed in the displaywhen the user's head is turned in the direction of the physical basestation. In some alternative embodiments, the processor can provideinstructions to the HMD which result in the virtual base station notbeing displayed when the user is, for example, more than a shortpreselected distance away from the physical base station. (As above, thepreselected distance can be pre-programmed into the processor, or mayalternatively be selectable by the user.) In such cases, as above it maybe desirable to allow the user to perform some action on the physicalcontroller(s) which causes the processor to indicate where the physicalbase station is located, either by displaying the virtual base station210 in the HMD display again, or possibly by providing arrows in the HMDdisplay which indicate the direction the user should look or move toreach the physical base station.

While the above example uses two physical controllers, in someembodiments there may be only a single physical controller. In such acase, the virtual base station 210 and virtual controller 212 may nolonger be displayed in the virtual world when the single physicalcontroller has been removed from the physical base station. Otherchanges to the described method will be apparent to one of skill in theart.

FIG. 9 is a flowchart of a method implemented by a processor that mightresult in the virtual world displays shown in FIG. 2 through 7 aboveaccording to one embodiment. It is assumed that, as above, a user wearsa HMD having a display, a processor provides instructions that directwhat appears on the display, the HMD has a tracker used to track theposition and orientation of the HMD and thus of the user's head, andthere is a base station holding one or more controllers, each alsohaving a tracker. It is also assumed that, if, for example, theelectromagnetic field source or camera used to perform tracking of theHMD and controller trackers is not located in the base station, there isalso a tracker in the base station so that the processor is able todetermine the position and orientation of the base station, and thustrack the position and orientation of the HMD and controller(s) relativeto the base station.

At step 902, the processor generates and outputs instructions to the HMDto display on the HMD display a generated virtual world. At step 904,the processor determines the position and orientation of the HMD fromthe tracker in the HMD, and continues to do so at regular intervals.

At step 906, the processor generates and outputs instructions to the HMDto display on the HMD display a virtual base station, one or morevirtual controllers placed or docked on the virtual base station, and avirtual hand or hands, such as is shown in FIG. 3. As above, at thispoint the virtual hand will generally be in an assumed location basedupon the position and orientation of the tracker in the HMD, and isprovided to help guide the user in locating the physical base stationwith the user's physical hand.

At step 908, the processor determines based upon the position andorientation of the tracker in the HMD (and the tracker in the basestation if appropriate) that the user has moved toward the physical basestation. In response, at step 910 the processor generates and outputsinstructions to the HMD to display on the HMD display the virtual basestation, the virtual controller still placed or docked on the virtualbase station, and the virtual hand, but now with the virtual hand closerto the virtual base station, for example as shown in FIG. 4.

At step 912, the processor determines that the physical controller (ormore than one of them, if there are more than one) has been removed fromthe physical base station, such as is shown in FIGS. 5 and 6. As above,the processor may be able to detect that an electrical connectionbetween the physical base station and the physical controller has beenbroken, such as a connection that allows batteries in the physicalcontroller to be recharged by circuitry in the physical base station.Alternatively, the physical controller may also have a tracker thatallows the processor to determine the position and orientation of thephysical controller, and, once the processor has determined that thedistance between the physical base station and the physical controllerexceeds a certain distance, the processor can therefore determine thatthe physical controller has been removed from the physical base station.

Finally, at step 914 the processor generates and outputs instructions tothe HMD to display on the HMD display the virtual hand, but not thevirtual base station or virtual controller, as shown in FIG. 6. Asabove, the processor can change the instructions so as to no longerdisplay the virtual controller when the physical controller has movedmore than a preselected distance from the physical base station, asdetermined from the position and orientation of the tracker in thephysical controller.

It will be apparent to one of skill in the art that some of these stepsmay be performed in a different order than specified, while othersshould be performed in order to obtain the desired effect. For example,the processor may determine the position and orientation of the HMDtracker before generating and outputting the instructions to the HMD todisplay the generated virtual world. On the other hand, it will mostlikely not be appropriate to send the instructions that result in theHMD no longer displaying the virtual controller before the user hasremoved the physical controller from the physical base station.

As above, the physical controller has a tracker, so the virtual hand maynow be in an apparent location corresponding to the actual location ofthe user's physical hand rather than in an assumed location, since theprocessor will know where the user's physical hand is from the trackerin the physical controller. The ability of the user to see virtual handsin the locations corresponding to the user's physical hands can help theuser to believe in the reality of the virtual world, as well as help inpicking up the physical controllers as described above. As explainedelsewhere herein, this can also help the user to operate in a selectedarea, thereby avoiding physical obstacles in the real world.

In this way, the display of the virtual base station, virtualcontroller(s) and virtual hand(s) allows the user to locate and pick upthe physical controller(s) primarily by sight rather than by touch, eventhough the user is already wearing the HMD and seeing only the virtualworld, and is thus unable to see the physical controller(s). Reversingthe process as explained above similarly allows the user to return thephysical controller(s) to the base station when the user is ready toexit the virtual world, or for other reasons, without having to removethe HMD first.

As above, another issue that may arise when the user is in the virtualworld is that the user may not be able to see the surrounding physicalenvironment and therefore may not be aware of how far the user has movedin the physical world, and may collide with physical obstacles whiletrying to accomplish tasks in the virtual world. For example, if a useris physically located in the user's living room and playing a gamewithin a virtual world, there will likely be physical objects orobstacles present in the room such as a sofa or other furniture, lamps,and even the walls of the room. Since the display in the HMD blocks theuser's view of the physical world, the physical objects are not visibleto the user. Further, since it is generally considered preferable topresent the user with a view of only the virtual world in order to getthe desired immersive effect, these physical items are similarly notpresented to the user through the HMD display (other than the virtualbase station 210 and virtual controllers 212 for the purposes explainedabove). However, if the user cannot tell where the user is, or wherethese physical objects are, while the user in the virtual world, thereis a risk that the user's motions may cause him or her to collide withthese objects, possible resulting in damage to the objects or eveninjury to the user.

Thus, it is desirable to provide a means by which the user may be madeaware that the user is approaching a boundary of a selected area inwhich the user desires to remain, to avoid physical objects or for anyother reason, without distracting the user from the immersive virtualworld. In one embodiment, a selected area having an outer boundary isdetermined, such as a “safe” area in which the user can move freelywithout coming into contact with the physical objects in the user'svicinity. The selected area may already be clear of physical obstacles,or the user or other person may move physical objects to clear theselected area. (Some objects, such as rugs, may not need to be moved.)The processor provides instructions to the HMD display to include adisplay of a virtual fence at the boundary when the user comes within apreselected distance of the boundary of the selected area, thusproviding a warning that the user is approaching the boundary of theselected area.

FIG. 10 shows a representation of a user 1002 in a virtual space. (Asthe display is usually presented from the user's point of view, the user1002 does not see himself or herself, but is shown here for illustrativepurposes only.) As above, the processor instructs the HMD display toshow the user a virtual world, as illustrated here a room including aback wall 1004, side walls 1006, ceiling 1008 and floor 1010. Also asabove, the user will see his or her virtual hand 1012; again, virtualhand 1012 will be in the location of the user's actual hand if the user1002 is holding a controller having a tracker that provides position andorientation information to the processor, and in an assumed location ifthe user 1002 is not holding such a controller. As above, since the useris wearing a HMD that displays the virtual world, the physical world andany physical objects are not visible to the user.

As above the processor is tracking the position and orientation of theuser's head from the tracker of the HMD, and possibly tracking theposition and orientation of the user's hand if the user is holding acontroller with a tracker. Thus, the processor knows where the user'shead and hands are located; if the processor has been provided withinformation about the boundary of the selected area then the processorcan also determine when the user's head and/or hand approaches theboundary.

In one embodiment, when the user's head and/or hand approaches to withina preselected distance, for example, one foot (again, eitherpreprogrammed into the processor or user selectable), of the boundary ofthe selected area the processor generates a warning that the user isclose to the boundary so that the user can take action to avoid theboundary and thus potential collision with the physical objects. In someembodiments, the processor can generate this warning by generatinginstructions to the HMD display to add some virtual element to thevirtual world which can be seen by the user.

FIG. 11 shows one such form of visual warning. In FIG. 11 it is assumedthat the user 1002 is holding a controller with a tracker in the handcorresponding to virtual hand 1012, and that the user 1002 has extendedthat hand to a point within the preselected distance from the boundaryof the selected area. The display in FIG. 11 is similar to that of FIG.10, except that since the user 1002 has extended his hand holding thecontroller, the virtual hand 1012 now extends further in front of theuser 1002.

In addition, since the hand of the user 1002 is within the preselecteddistance of the boundary of the selected area, the processor generatesinstructions to the HMD to generate the visual warning. As illustratedin this embodiment, this visual warning is in the form of a grid oflines or a “virtual fence” located at the boundary, as indicated in FIG.11 by the dashed lines 1114. The grid can be glowing or colored ifdesired to be more visible to the user. In other embodiments, othershapes than a grid-shaped virtual fence can be displayed to the user.For example, the developer of a particular application may choose someother design to provide contrast with, or maintain thematic correctnesswith, the particular virtual world being displayed. In some embodiments,even a uniform translucent glow of a particular color, such as red, maybe appropriate to serve as the visual warning. In addition, thebrightness of the visual warning may increase as the distance the useris from the boundary decreases to provide further notice that the useris getting still closer to the boundary.

The virtual fence 1114 need not cover the entire display of the HMD; itis believed that it will generally be sufficient for the processor togenerate instructions to have the HMD display show the virtual fence1114 in an area around the point in the virtual world at which theuser's head or hand has approached the boundary of the selected area. Tominimize rendering cost, the fence may be rendered as a projectedtexture on small pieces of 3D geometry that cover only the areas wherethe fence is to be visible. A cylindrical projection is calculated byconverting the position of the 3D geometry into cylindrical coordinates,which are used to offset a texture map. This gives the impression thatthe grid lines of the fence are stationary as the 3D geometry is movedto follow the tracker while it is in the proximity of the boundary,minimizing screen coverage of the geometry for optimal processorperformance. Blending is used to create a smooth fade transition on theedges of the geometry, and to make it more transparent as the trackingdevice moves away from the boundary. When the tracker is beyond thepreselected distance, the grid becomes completely transparent and the 3Dgeometry is removed completely to save processing time.

In addition, a different type of visual warning may be appropriate whenthe user is not looking in the direction in which he or she isapproaching the boundary of the selected area. For example, if a user isapproaching the boundary of the selected area but not looking in thedirection of the boundary, if the HMD display is instructed to provideonly a view of the virtual world in front of the user there may be novisual warning. Thus, in these cases, some other type of visual warning,perhaps with directional arrows indicating the direction of the boundaryin relation to the user's view, may be appropriate. It is also possibleto include auditory warnings instead of, or in addition to, a visualwarning.

As above, the fence is displayed in the vicinity of the user's head orhands when the user's head or hands are approaching the boundary asdetermined by the trackers on the user's head or hands. This may requirean earlier warning when it is the user's head that approaches theboundary, since this likely means that the user's body is close to theboundary, and thus that the user's hands may easily go beyond theboundary as a result of an intentional motion or even a reflexivemotion. Thus, the preselected distance within which an approach by theuser causes the processor to display the visual warning may be definedseparately for the HMD tracker and the handheld controller tracker(s).In addition, the preselected distance may vary by application, so that,for example, it is greater in an application in which larger bodymovements are expected from the user than in an application in whichsuch body movements are not expected.

If the user's head or hands go beyond the boundary as determined by thetrackers, the processor can provide additional messaging or warning,such as text or auditory warnings, or an additional visual warning, thatpersists until the user returns to the selected area within theboundary. In addition, if the user's head goes beyond the boundary, theprocessor can cause the HMD to fade to black, and present, for example,only a warning message and/or a beacon or arrows directing the user tomove in the direction of the boundary and the selected area.

The boundary of the selected area may be determined in several ways. Insome embodiments, a person, such as the user or another person, forexample the parent of a child user, may input information into theprocessor indicating that there is a circular area in front of the basestation and having a measured diameter that is clear of objects. FIG. 12illustrates this configuration.

In one embodiment, the information input to the processor is determinedby activating one of the trackers, either in the HMD or a handheldcontroller, at a position 1204 directly in front of a base station 1202.From the position 1204 of the tracker (and the position of a basestation tracker if, for example, the electromagnetic field source orcamera are not in the base station), the processor can determine how faraway the tracker is from the base station 1202, and then determine acircular area 1206 having a radius of that distance centered at thatpoint 1204. In this case, the user should check to see that there are noobstacles within that defined circle since the processor has no way todetermine whether there are any such obstacles in the circular area1206.

In another embodiment, the radius (or diameter) of the circular area1206 is measured (for example, by the user), and the processor receivesas input the measurement. The processor may also receive the fact thatthe circular area is directly in front of the physical base station 1204(again using information from a base station tracker if appropriate), oralternatively the user may be told to set up the physical base station1204 and circular area 1206 in this fashion. In some embodiments, suchinstructions to the user could even include the radius (or diameter) ofa preprogrammed circular area, and direct the user to clear an area ofsuch size of any obstacles.

In some cases, due to the layout of a particular space, or the objectsin the space, it may not be convenient or desirable to locate a desiredcircular selected area immediately in front of the base station as inFIG. 12. Thus, as shown in FIG. 13, in one alternative embodiment, theradius (or diameter) of an open circular area 1306 is measured, e.g., bythe user, and that measurement is received by the processor as input. Atracker is placed at the center of the circular area, and by determiningthe tracker's position and orientation the processor is able todetermine the offset distance and direction to the center of thecircular area from either the base station or another location havingthe electromagnetic field source, camera or other tracking components.This allows the processor to define the circular area having themeasured radius, and centered at the measured offset position, as aselected area.

In still another embodiment, an irregularly shaped area is defined, asshown in FIG. 14. A boundary 1404 around an irregularly shaped area,which is not circular, can be defined, for example, by having the userwalk around a perimeter chosen by the user with one of the trackers todefine the outer boundary of a selected area, for example, one in whichthere are no obstacles. The processor determines the position of thetracker at a number of sampling points as the user moves around thechosen perimeter, and from those points determines the boundary 1404.This approach may be desirable when the size of a circle 1406 that canbe inscribed within boundary 1404 is limited by nearby objects 1408, butthere is some additional room between objects 1408 in which the user canbe free to move without risk of collision with objects 1408.

FIG. 15 is a flowchart of a method implemented by a processor that mightresult in the display of a visual warning such as the virtual fence 1114of FIG. 11, or other warnings as described above. As in the method oflocating objects described above, it is assumed that a user wears a HMDhaving a display, a processor provides instructions that direct whatappears on the display, and the user is holding a handheld controllerhaving a tracker which can be used to track the position and orientationof the handheld controller.

At step 1502, the processor receives as an input information from whichthe processor can define a boundary surrounding a selected area, forexample, one in which there are no physical obstacles. As above, thisinformation will typically be entered by a user or someone else, or mayin part or whole be received from a tracker, and may be the radius andlocation of the center of a circle of a determined diameter that isclear of obstacles, or positions around an irregularly shaped areaprovided by the user or other person walking around the boundary holdinga tracker. Alternatively, as above, the user may be told to clear anyobstacles from a circular area of a preprogrammed radius or diameter.

At step 1504, the processor determines the position of the handheldcontroller from the controller tracker. Next, at step 1506, theprocessor generates and outputs instructions to the HMD to display onthe HMD display a generated virtual world.

Next, at step 1508, the processor determines from the position of thehandheld controller that the handheld controller is within a preselecteddistance of the boundary. Finally, at step 1510 the processor generatesand outputs instructions to the HMD to display on the HMD a visualwarning to the user. As above, this visual warning can be in the form ofgrid lines such as a virtual fence, or some other configuration. Inappropriate situations an audible warning can be generated in additionto, or in lieu of, the visual warning.

As with the method of assisting a user in locating a physical controllerabove, a person of skill in the art will appreciate that some of thesteps of the method of assisting a user in avoiding physical objects maybe performed in a different order, while certain steps will be mostadvantageously performed in a particular order.

Also as above, the user is wearing a HMD that may also have a tracker.In such a case, the processor may also display a visual warning when theuser's head approaches the boundary. Since the user does not see his orher own head in the virtual world, the visual warning in this case may,for example, again be in the form of grid lines, and appear in thevirtual world to be immediately in front of the user.

Also as above, in various embodiments, the processor can generate thevisual warning in the form of grid lines or a virtual fence, or in anyother desired shape or configuration. In some embodiments, it may againbe appropriate to generate an audio warning as well, or instead of, thevisual warning.

By alerting a user in this fashion that the user is approaching theboundary of the selected area, the described system and method allows auser to take action to avoid the boundary and stay within the selectedarea defined by that boundary. This can assist the user in avoidingcollision with physical objects in the area of the user, and anypossible resulting damage to such objects or injury to the user.

The disclosed system and method has been explained above with referenceto several embodiments. Other embodiments will be apparent to thoseskilled in the art in light of this disclosure. Certain aspects of thedescribed method and apparatus may readily be implemented usingconfigurations or steps other than those described in the embodimentsabove, or in conjunction with elements other than or in addition tothose described above. It will also be apparent that in some instancesthe order of steps described herein may be altered without changing theresult of performance of all of the described steps.

For example, it is expected that the described apparatus may beimplemented in numerous ways. As noted above, the controller(s) may nestor dock in a base station, or may be separate handheld controllersfamiliar to users of video game consoles. As above, various embodimentsmay be applicable to HMDs that block the user's view of all or only partof the physical world. Even some uses of a device such as Google Glass,which obscures only a small portion of the user's field of vision of thephysical world, may benefit from some embodiments described herein.

There may be a single processor, or multiple processors performingdifferent functions of the functions described herein. As above, aprocessor may be located in the base station, or in a separate location,or even in the HMD or one of the controllers if desired. One of skill inthe art will appreciate how to determine which and how many processorswill be appropriate for a specific intended application. Similarly, asexplained above, various tracking techniques may be used in addition toelectromagnetic tracking, and in some embodiments some of the othercomponents used for tracking, such as an electromagnetic field source ora camera, may be located in the base station, while in other embodimentsthey may be located elsewhere.

One of skill in the art will also appreciate that controllers may be ofvarious shapes other than those shown or described herein. It may bedesirable to have a controller in the shape of an object that isintended to appear in the virtual world so that the user's hands “feel”the same object that is seen in the HMD; for example, a controller maytake the shape of a bat, tennis racket, golf club or other sportingimplement (or the handle of such an object), a gun, sword or otherweapon (or again the handle thereof), or some other toy or device whichwill appear in the virtual world.

These and other variations upon the embodiments are intended to becovered by the present disclosure, which is limited only by the appendedclaims.

What is claimed is:
 1. A method of assisting a user wearing ahead-mounted display (HMD) in locating a physical handheld controllerlocated on a physical base station, the HMD covering the user's field ofvision and having a HMD tracker which provides information regarding theposition of the HMD, the method comprising: generating and outputting,by a processor, instructions to the HMD to display on the HMD a virtualworld; determining, by the processor, the position of the HMD in thephysical world using information from the HMD tracker; generating andoutputting, by the processor, instructions to the HMD to display on theHMD a virtual base station, a virtual controller and a virtual hand inthe virtual world, the virtual base station and virtual controller inapparent locations corresponding to actual physical locations of thephysical base station and physical controller and the virtual hand in anassumed apparent location; determining, by the processor, based upon theposition of the HMD, that the user has moved toward the physical basestation in the physical world; generating and outputting, by theprocessor, instructions to the HMD to display on the HMD the virtualbase station, virtual controller, and virtual hand, with the virtualhand closer to the virtual base station than the assumed apparentlocation; determining, by the processor, that the physical controllerhas been removed from the physical base station; and generating andoutputting, by the processor, based upon the determination that thephysical controller has been removed from the physical base station,instructions to the HMD to display on the HMD a virtual hand in thevirtual world, without displaying the virtual base station or thevirtual controller.
 2. The method of claim 1, wherein the physical basestation has a base station tracker which provides information regardingthe position of the physical base station, and wherein generating andoutputting instructions to the HMD to display on the HMD a virtual basestation is based in part upon the information from the base stationtracker.
 3. The method of claim 1, wherein the physical base station hasa base station tracker which provides information regarding the positionof the physical base station, and wherein determining that the user hasmoved toward the physical base station is additionally based upon theinformation from the base station tracker.
 4. The method of claim 1wherein determining that the physical controller has been removed fromthe physical base station is accomplished by determining that anelectrical connection between the physical controller and the physicalbase station has been broken.
 5. The method of claim 1 whereindetermining that the physical controller has been removed from thephysical base station is accomplished by determining that the positionof the HMD has been moved beyond a preselected distance from thephysical base station.
 6. The method of claim 1 wherein the physicalcontroller further comprises a controller tracker which providesinformation regarding the position of the physical controller, andwherein determining that the physical controller has been removed fromthe physical base station is accomplished by determining, by theprocessor using information from the controller tracker, that theposition of the physical controller has been moved beyond a preselecteddistance from the physical base station.
 7. The method of claim 1wherein generating and outputting instructions to the HMD to display onthe HMD the virtual hand in the virtual world, without displaying thevirtual base station or the virtual controller is accomplished bydetermining that the HMD has been moved beyond a preselected distancefrom the physical base station.
 8. The method of claim 1 wherein thephysical controller further comprises a controller tracker whichprovides information regarding the position of the physical controller,and wherein generating and outputting instructions to the HMD to displayon the HMD the virtual hand in the virtual world, without displaying thevirtual base station or the virtual controller, is accomplished bydetermining, by the processor using information from the controllertracker, that the position of the physical controller has been movedbeyond a preselected distance from the physical base station.
 9. Asystem for use with a head-mounted display (HMD), the HMD covering auser's field of vision and having a HMD tracker which providesinformation regarding the position of the HMD, comprising: a physicalbase station; a physical handheld controller located on the physicalbase station; and a processor configured to: generate and outputinstructions to the HMD to display on the HMD a virtual world; determinethe position of the HMD in the physical world using information from theHMD tracker; generate and output instructions to the HMD to display onthe HMD a virtual base station, a virtual controller and a virtual handin the virtual world, the virtual base station and virtual controller inapparent locations corresponding to actual physical locations of thephysical base station and physical controller and the virtual hand in anassumed apparent location; determine, based upon the position of theHMD, that the user has moved toward the physical base station in thephysical world; generate and output instructions to the HMD to displayon the HMD the virtual base station, virtual controller, and virtualhand, with the virtual hand closer to the virtual base station than theassumed apparent location; determine that the physical controller hasbeen removed from the physical base station; and generate and outputinstructions to the HMD, based upon the determination that the physicalcontroller has been removed from the physical base station, to displayon the HMD the virtual hand in the virtual world, without displaying thevirtual base station or the virtual controller.
 10. The system of claim9, wherein the physical base station has a base station tracker whichprovides information regarding the position of the base station, and theprocessor is further configured to determine the position of thephysical base station in the physical world using the information fromthe base station tracker, and the instructions to the HMD to display onthe HMD a virtual base station are based in part upon the position ofthe physical base station.
 11. The system of claim 9, wherein thephysical base station has a base station tracker which providesinformation regarding the position of the base station, the processor isfurther configured to determine the position of the physical basestation in the physical world using the information from the basestation tracker, and the determination that the user has moved towardthe physical base station is additionally based upon the position of thephysical base station.
 12. The system of claim 9 wherein the processoris further configured to determine that the physical controller has beenremoved from the physical base station by determining that an electricalconnection between the physical controller and the physical base stationhas been broken.
 13. The system of claim 9 wherein the processor isfurther configured to determine that the physical controller has beenremoved from the physical base station by determining that the positionof the HMD has been moved beyond a preselected distance from thephysical base station.
 14. The system of claim 9 wherein the physicalcontroller further comprises a controller tracker which providesinformation regarding the position of the physical controller, and theprocessor is further configured to determine, using information from thecontroller tracker, that the physical controller has been removed fromthe physical base station by determining that the position of thephysical controller has been moved beyond a preselected distance fromthe physical base station.
 15. The system of claim 9 wherein theprocessor is further configured to generate and output instructions tothe HMD for display on the HMD the virtual hand in the virtual world,without displaying the virtual base station or the virtual controllerwhen the processor determines that the HMD has been moved beyond apreselected distance from the physical base station.
 16. The system ofclaim 9 wherein the physical controller further comprises a trackerwhich provides information regarding the position of the physicalcontroller, and the processor is further configured to generate andoutput instructions to the HMD for display on the HMD the virtual handin the virtual world, without displaying the virtual base station or thevirtual controller, when the processor determines, using informationfrom the controller tracker, that the physical controller has been movedbeyond a preselected distance from the physical base station.
 17. Anon-transitory computer-readable medium having embodied thereon aprogram, the program being executable by a processor to perform a methodof assisting a user wearing a head-mounted display (HMD), the HMDcovering the user's field of vision and having a tracker which tracksthe position of the HMD, in locating a physical handheld controller thatis initially placed on a physical base station, the method comprisingthe steps of: generating and outputting, by a processor, instructions tothe HMD to display on the HMD a virtual world; determining, by theprocessor, the position of the HMD in the physical world usinginformation from the HMD tracker; generating and outputting, by theprocessor, instructions to the HMD to display on the HMD a virtual basestation, a virtual controller and a virtual hand in the virtual world,the virtual base station and virtual controller in apparent locationscorresponding to actual physical locations of the physical base stationand physical controller and the virtual hand in an assumed apparentlocation; determining, by the processor, based upon the position of theHMD, that the user has moved toward the physical base station in thephysical world; generating and outputting, by the processor,instructions to the HMD to display on the HMD the virtual base station,virtual controller, and virtual hand, with the virtual hand closer tothe virtual base station than the assumed apparent location;determining, by the processor, that the physical controller has beenremoved from the physical base station; and generating and outputting,by the processor, based upon the determination that the physicalcontroller has been removed from the physical base station, instructionsto the HMD to display on the HMD the virtual hand in the virtual world,without displaying the virtual base station or the virtual controller.